Separate Modifiability and the Search for Processing Modules

One approach to understanding a complex process or system starts with an attempt to divide it into modules: parts that are independent in some sense, and functionally distinct. In this chapter I discuss a method for the modular decomposition of neural and mental processes that reflects recent thinking in psychology and cognitive neuroscience: This process-decomposition method, in which the criterion for modularity is separate modifiability, is contrasted with task comparison and its associated subtraction method. Five illustrative applications are based on the event-related potential (ERP), transcranial magnetic stimulation (TMS), and functional magnetic resonance imaging (fMRI). * E-mail: [email protected] Tel: 215-898-7162 Fax: 215-898-7210 Address: Department of Psychology University of Pennsylvania 3815 Walnut Street Philadelphia, PA 19104-6196, USA Sternberg A&P XX: Separate Modifiability & Modules 11/6/02 Page 2 Separate Modifiability and the Search for Processing Modules Saul Sternberg 1. Modules and Modularity The first step in one approach to understanding a complex process or system is to attempt to divide it into modules: parts that are independent in some sense, and functionally distinct.1 2 In the present context the complex entity may be a mental process, a neural process, the brain (an anatomical processor), or the mind (a functional processor). Four corresponding senses of "module" are: Module1: A part of a mental process, functionally distinct from other parts, and investigated with behavioral measures, supporting a functional analysis. Module2: A part of a neural process, functionally distinct from other parts, and investigated with brain measures, supporting a neural-process analysis. Module3: A neural processor 3 (part of the brain), a population Pα of neurons that is functionally specialized to implement a particular neural process α . If it is also localized (the sole occupant of a delimited brain region) and the only population that implements α , one may find selective task impairment from localized damage of Pα , and selective activation of Pα by tasks that require α .4 Module4: A mental processor or faculty (part of the mind), functionally specialized ("domain specific"), informationally isolated from (some) other processors ("encapsulated"), and a product of evolution. (Spelke, this volume, Chapter XX.) Most of the present discussion will be concerned with the first two senses. In what follows I describe and illustrate an approach to the decomposition of mental and neural processes into Modules1 and Modules2 that reflects recent thinking in psychology and cognitive neuroscience. 5 This process-decomposition method and three illustrations based on the event-related potential (ERP) are described in Section 2. I contrast process decomposition with the more familiar taskcomparison method in Section 3, describe an example of the latter based on the effects of repetitive transcranial magnetic stimulation (rTMS), and discuss the subtraction method as an embodiment of task comparison. Unlike task comparison, which is often used in a way that requires modularity to be assumed without test (Shallice, 1988, Ch. 11; Sternberg, 2001, Appendix A.1.), the process-decomposition method incorporates such a test. I consider the use of fMRI activation maps in process decomposition for the discovery of Modules2 in Section 4, list 1. Heuristic arguments for the modular organization of complex biological computations have been advanced by Simon (1962) and, in his "principle of modular design", by Marr (1976). 2. A module may itself be composed of modules. 3. Processes occur over time; their arrangement is described by a flow-chart. They are often confused with processors, which are parts of a machine, and whose arrangement is described by a circuit diagram. 4. By "selective impairment" I mean impairment of tasks that require α and not of tasks that don’t. By "selective activation" I mean activation of Pα by tasks that require α and not by tasks that don’t. In practice, selective activation is sometimes taken to mean the weaker differential activation (Kanwisher, Downing, Epstein, & Kourtzi, 2001), akin to the tuning curves for simple features. 5. In Sternberg (2001) I discuss and defend the method, describe its antecedents, illustrate it with a dozen applications to mental and neural processes, and explicate its inferential logic. Sternberg A&P XX: Separate Modifiability & Modules 11/6/02 Page 3 some desiderata in such applications, and present an example. I briefly consider the question of task-general modules (Section 5) and the relation between Modules2 and Modules3 (Section 6), and close with a few questions (Section 7). 2. The Process-Decomposition Method 2.1 Separate Modifiability Much thinking by psychologists and brain scientists about the decomposition of complex processes appeals implicitly to separate modifiability as a criterion for modularity: Two (sub)processes A and B of a complex process (mental or neural) are modules iff each can be changed independently of the other. To demonstrate separate modifiability of A and B, we must find experimental manipulations (factors) F and G that influence them selectively, i.e., such that A is influenced by F but is inv ariant with respect to G, whereas B is influenced by G but is inv ariant with respect to F.6 Such double dissociation of subprocesses should be distinguished from the more familiar double dissociation of tasks (Sternberg, In Press). 2.2 Processes and Their Measures, Pure and Composite How do we demonstrate that a process is influenced by a factor, or inv ariant with respect to it? We know only about one or more measures MA of process A, not about the process as such. Depending on the available measures, there are two ways to assess separate modifiability of A and B. Suppose we have pure measures MA and MB of the hypothesized modules: A pure measure of a process is one that reflects changes in that process only. Examples include the sensitivity and criterion parameters of signal-detection theory (reflecting sensory and decision processes), and the durations of two different neural processes. To show that F and G influence A and B selectively, we must demonstrate their selective influence on MA and MB. The influence and invariance requirements are both critical. It is unfortunately seldom appreciated that persuasive evidence for invariance cannot depend solely on failure of a significance test of an effect: such a failure could merely reflect variability and low statistical power.7 Instead of pure measures, suppose we have a composite measure MAB of the hypothesized modules — a measure to which they both contribute. To demonstrate selective influence in this case we must also know or confirm a combination rule — a specification of how the contributions of the modules to the measure combine. Examples of composite measures are the ERP at a particular point on the scalp (which may reflect several ERP sources), and mean reaction time, RT (which may depend on the durations of several processes). Whereas factorial experiments are desirable for pure measures8 , with a composite measure they are essential; unfortunately they are rare. A giv en measure may be pure or composite, depending on the hypothesized modules of interest. However, rather than being determined a priori, this attribute of a measure is one of the 6. Separate modifiability of A and B is also evidence for their functional distinctness (Sternberg, 2001, p. 149). And information about what a process does is provided by the sets of factors that do and don’t influence it. 7. When an effect is claimed to be null it is important in evaluating the claim to have at least an index of precision (such as a confidence interval) for the size of the effect. An alternative is to apply an equivalence test (Berger & Hsu, 1996) that reverses the asymmetry of the standard significance test. In either case we need to specify a critical effect size (depending on what we know and the particular circumstances) such that it is reasonable to treat the observed effect as null if, with high probability, it is less than that critical size. 8. See Sternberg (2001), Appendix A.9. Sternberg A&P XX: Separate Modifiability & Modules 11/6/02 Page 4 components of a theory that is tested as part of the process-decomposition method.9 2.3 Three Examples of Decomposition of Neural Processes with ERPs Here I provide brief summaries of three applications of these ideas, in which the brain measures are derived from ERPs.10 Parallel Modules for Selecting a Response and Deciding Whether to Execute It. Osman, Bashore, Coles, Donchin, and Meyer (1992) investigated a task in which a speeded response was required to two of four equiprobable stimuli. The location of the stimulus (left vs right) indicated the correct response (left-hand vs right-hand); its category (letter vs digit) determined whether that response should be executed (Go vs NoGo trials). The two factors were the stimulus-response mapping (SRM , spatially compatible vs incompatible), and Go-NoGo (letter-digit) discriminability (GND, easy vs hard). The two hypothesized pure measures depend on the lateral asymmetry of the motor-cortex voltage versus time for Go and NoGo trials, AMC (t, Go) and AMC (t, NoGo). 11 One measure, (Mα ) is the time interval from stimulus onset to when the sum AMC (t, Go) + AMC (t, NoGo) exceeds zero, which reflects the duration of α , the hypothesized response-selection module. The other measure (Mβ ) is the interval from stimulus onset to when the difference AMC (t, Go) − AMC (t, NoGo) exceeds zero, which reflects the duration of β , the hypothesized execution-decision module. They found that Mα is influenced by SRM (effect, ∆ = 121 ± 17 ms, n = 6) but negligibly by GND (∆ = 2. 5 ± 5. 0 ms, n = 6)12 , whereas Mβ is influenced by GND (∆ = 43 ± 14 ms) but negligibly by SRM (∆ = 3. 3 ± 8. 8 ms), evidence for the separate modifiability of α and β .13 Other aspects of the data indicate that α and β operate in parallel. Serial Modules for Interpreting a Stimulus and Initiating the Response. Smulders, Kok, Kenemans, and Bashore (1995) investigated a task in which a digit stimulus indicated which hand had to execute a speeded response. The two factors were stimulus quality (SQ, two lev els) and response complexity (RC, a single keystroke vs a string of three keystrokes). The two hypothesized pure measures were Mα , the duration of process α (from the stimulus to the onset of motor cortex asymmetry), and Mγ , the duration of process γ (from the onset of motor cortex asymmetry to the response). They found that Mα is influenced by SQ (∆ = 34 ± 6 ms, n = 14) but negligibly by RC (∆ = 4 ± 8 ms), whereas Mγ is influenced by RC (∆ = 21 ± 7 ms) but negligibly by SQ (∆ = 1 ± 8 ms), evidence for two neural processing modules arranged as stages.14 9. See Sternberg (2001), Sections 2 and 3 and Appendix A.2.3. 10. These examples are treated in detail in Sternberg (2001) in Sec. 6, Appendix A.6, and Sec. 14. 11. AMC is the amplitude difference between the scalp ERPs associated with the parts of the motor cortex that control the left and right hands, taken in the direction that favors the response signaled by the stimulus location; an increase in AMC from baseline is sometimes called the lateralized readiness potential. 12. A cautionary note on the meaning of "negligible", using this example: With SE = 5.0 ms, 5 df , and a mean of 2.5 ms, a 95% confidence interval based on the t-statistic indicates that the GND effect may be as large as 15 ms. Another way of indicating the precision of the data is that a GND effect would have to be as large as ±13 ms to be detected, in the sense of differing significantly from zero at the p = 0. 05 level. See Note 7. 13. SE estimates are based on between-subject variability. Howev er, the SE values provided for the second and third examples are likely to be overestimates because balanced effects, such as those of condition order, were treated as error variance. Sternberg A&P XX: Separate Modifiability & Modules 11/6/02 Page 5 Two Modules in Word Classification. Kounios (1999, 2002) required subjects to classify each of a sequence of spoken nouns by meaning. Most of the words required no response, while 5% were targets (names of body parts) that called for a manual response. The words consisted of primes and probes. The factors (two lev els each) were the semantic relatedness (REL) of the probe to the preceding prime, and the semantic satiation (SAT ) of that prime (number of immediate repetitions of the prime before the probe). The data were the ERPs elicited by the non-target probes at several locations on the scalp. For present purposes, define for each location a composite measure Mα β as the mean ERP amplitude at that location during the epoch from 600 to 800 ms after probe onset. Now consider the following theory, with three components: (a) Subprocesses: the complex process of recognizing the probe as a non-target contains (at least) two subprocesses, α and β , carried out by different neural processors, Pα and Pβ . (b) Selective Influence: α is influenced by SAT but not REL, whereas β is influenced by REL but not SAT . (c) Combination Rule: Each process is an ERP source; physics tells us that at any location the combination rule for sources is summation. It can be shown that this theory implies that the effects of SAT and REL on Mα β will be additive at all scalp locations. 15 Kounios found such additivity (mean main effects of REL and SAT were 1. 3 ± 0. 2μV and 2. 1 ± 0. 4μV, respectively, while the mean interaction contrast was 0. 01 ± 0. 3μV, n = 36), supporting the above theory and hence the modularity of α and β .16 Also, the topographies of the two effects (their relative sizes across locations) differ markedly, indicating different locations in the brain for Pα and Pβ . 3. Process Decomposition versus Task Comparison The cases above exemplify a process decomposition method whose goal is to divide the complex process by which a particular task is accomplished into modular subprocesses, a method that has been used to find Modules1 and Modules2. The factor manipulations are not intended to produce "qualitative" changes in the complex process (such as adding new operations, or replacing one operation by another) which may be associated with a change in the task, just "quantitative" ones that leave it inv ariant.17 The task-comparison method is a more popular approach to understanding the structure of complex processes. Here one determines the influence of factors on performance in different tasks, rather than on different parts of the complex process used to carry out one task. The data pattern of interest is the selective influence of factors on tasks, i.e., the single and double dissociation of tasks. (A classical factor used in brain studies is the amount, 14. SQ and RC also had additive effects on concurrently measured RT (a composite measure), consistent with their selectively influencing two functional modules, A and C, that are arranged as stages. Together with the similarity of effect sizes in the neural and behavioral analyses, this suggests that A and C are implemented by α and β , respectively. 15. In the present context, the effect of a factor is the change in the measure produced by changing the level of that factor. Letting subscripts 1 and 2 indicate factor levels and letting (i, j) indicate the levels of RELi and SAT j , additivity (non-interaction) of the effects of SAT and REL means that Mα β (2, 2) − Mα β (1, 1) = [Mα β (2, 1) − Mα β (1, 1)] + [Mα β (1, 2) − Mα β (1, 1)]. 16. Support for the theory is support for all of its three components. However, because the combination rule is given by physics in this application, there is no need to test component (c). 17. Qualitative task changes should be avoided because they reduce the likelihood of discovering modules. Evidence is required to assert qualitative task invariance. One kind of evidence is the pattern of factor effects: For each factor, each change in level should influence the same operations and leave the same other operations invariant. The usefulness of such evidence is one of several reasons for using factors with more than two lev els. Unfortunately few studies (and none of the three examples above) hav e done so. See Sternberg (2001), Appendices A.2.1 and A.9.2. Sternberg A&P XX: Separate Modifiability & Modules 11/6/02 Page 6 usually presence vs absence, of damage in a particular brain region.) Although it may achieve other goals, task comparison is inferior to process decomposition for discovering the modular subprocesses of a complex process: The interpretation of task comparison often requires assuming a theory of the complex process in each task (specification of at least the set of subprocesses) and a theory of their relationship (which subprocesses are identical across tasks); the method includes no test of such assumptions. In contrast, process decomposition requires a theory of only one task, and, as illustrated by the examples above, incorporates a test of that theory. 3.1 An Example of Task Comparison: Effects of Magnetic Brain Stimulation An elegant example of task comparison is provided by Pascual-Leone, Theoret, Merabet, Kauffmann, and Schlaug (this volume, Chapter YY) in their experiment on the effects of rTMS on subjective numerical scaling of two tactile perceptual dimensions, based on palpation of the same set of tactile dot arrays by the fingers of one hand. The two dimensions were distance (between dots), and roughness. Where rTMS had an effect, it reduced the sensitivity of scale values to differences among stimulus objects. One measure of this effect is the slope, b, of the linear regression of post-rTMS scale values on non-rTMS scale values. If there were no effect we would have b = 1. 0. The data indicate that performance in the roughness-judgement task is influenced by rTMS of the contralateral somatosensory cortex (rTMSS ; brs = 0. 79 ± 0. 07, p = 0. 02, n = 11) but negligibly by rTMS of the contralateral occipital cortex (rTMSO; bro = 0. 98 ± 0. 03), while performance in the distance-judgement task is influenced by rTMSO (bdo = 0. 84 ± 0. 07, p = 0. 04) but negligibly by rTMSS (bds = 0. 95 ± 0. 04), a double dissociation of the two tasks.18 Plausible theories of the two tasks would include modules for control of stimulus palpation (α d , α r ), generation of a graded perceptual representation (β d , β r ), and conversion of this percept into a numerical response (γ d , γ r ). Any or all of these might differ between the tasks. The striking findings indicate that the members of one or more of these pairs of processes depend on different regions of the cortex in the two tasks. If the task theories also included the assumptions that α d and α r are the same (α d = α r = α ), and that γ d and γ r are the same (γ d = γ r = γ ), then we could assert that it is β d and β r that are implemented by processors in different regions. Furthermore, the results would then also suggest that neither α nor γ is sensitive to either rTMSS or rTMSO, perhaps indicating that α and γ are implemented by processors in neither of the stimulated regions. To justify these conclusions, the task theories would of course have to be validated. 3.2 The Subtraction Method: Task Comparison with a Composite Measure One variety of task comparison, devised by Donders (1868) for the RT measure, has also often been used with brain activation measures (e.g., Petersen, Fox, Posner, Mintun, & Raichle, 1988). Suppose we are interested in studying a subprocess β of a complex process. If β were implemented by a localized neural processor Pβ in region Rβ , then the level of activation of Rβ might be a pure measure of the subprocess. Suppose instead, however, that Pβ is not localized (Haxby, this volume, Chapter ZZ), and all we have is a composite measure that reflects contributions from more than one subprocess. Under these conditions the subtraction method is sometimes used. This method requires three hypotheses. In a simple case they are: H1 (Task 18. Subscripts d and r refer to the two tasks; subscripts s and o refer to the two stimulated brain regions. SEs are based on between-subject variability. Also supporting the claim of double dissociation, the differences, bro − brs and bds − bdo are significant, with p = 0. 01 and p = 0. 04, respectively. Howev er, because non-rTMS measurements were made only before rTMS, rather than being balanced over practice, straightforward interpretation of the slope values requires us to assume negligible effects of practice on those values. Sternberg A&P XX: Separate Modifiability & Modules 11/6/02 Page 7 Theory 1): Task 1 is accomplished by process α ; H2 (Task Theory 2): Task 2 is accomplished by α and β ; H3 (Combination Rule): Contributions uα of α and uβ of β to measure Mα β combine by summation. (Possible justifications of this combination rule include, for a brain-activation measure, an assumption that α and β are implemented by different populations of neurons that contribute independently to the measure, and for an RT measure, an assumption that α and β are arranged as stages.) Let the Mα β measures in the two tasks be M1 and M2. The hypotheses imply that M1 and M2 − M1 are estimates of uα and uβ , respectively, and can thus play the roles of pure measures of α and β . But having these measures provides no test of the hypotheses.19 If summation proves to be incorrect as the combination rule, then other strategies may be available. For example, suppose measured activation were shown to be a decelerating function of the amount of neural activity, in particular, a logarithmic function. Then we would have Mα β = log(uα + uβ ), and the subtraction method could be applied to the transformed activation measure M ′ α β = exp(Mα β ) = uα + uβ . 4. Neural Processing Modules Inferred from Activation Maps Modular neural subprocesses can be discovered by applying process decomposition to the kinds of activation measures provided by PET and fMRI. Suppose localization of function, such that two such subprocesses, α and β , are implemented by different processors, Pα and Pβ , in nonoverlapping regions Rα and Rβ . Then activation levels in Rα and Rβ are pure measures of α and β , and, with sufficiently precise data and factors that influence the subprocesses selectively, separate modifiability is easy to test.20 If, however, α and β are implemented by different neural processors, Pα and Pβ (or by the same processor Pα β ) in one region, Rα β , then the activation level in Rα β is a composite measure that depends on both α and β , and to test separate modifiability we must know or show how their contributions to the activation measure are combined.21 4.1 Some Desiderata for Process-Decomposition using fMRI 1. The subject should be performing a task while measurements are taken. Even in sensory studies enough evidence has emerged favoring task effects at early levels of cortical processing so it is no longer appropriate merely to present stimuli to a passive observer.22 2. To increase the likelihood of discovering modules, the subject should be performing the same task as factor levels are varied. By "same task" I mean that a persuasive argument can be made that for all combinations of factor levels, the same set of processing operations is involved, varying only "quantitatively".23 19. One way to test the set of hypotheses is to extend it by finding two additional tasks that satisfy H4 (Task 3 is accomplished by α and γ ) and H5 (Task 4 is accomplished by α , β , and γ ) and to extend H3 by including γ . The extended set of hypotheses requires that M4 − M3 = M2 − M1. 20. Such tests require no assumptions about whether a change in factor level causes an increase or decrease in activation. This contrasts with the assumption, sometimes used to infer Modules3 (Kanwisher, et al., 2001), that stimuli more prototypical of those for which a processor is specialized will produce greater activation. 21. For example, if the combination rule is summation (often assumed without test) and if factors F and G influence α and β selectively, then the effects of F and G will be additive. Finding such additivity in a factorial experiment would support the combination rule as well as selective influence. If summation is assumed erroneously, selective influence would be obscured: the effect of each factor would appear to be modulated by the level of the other. 22. For example, with passive observing, different stimuli may attract attention differentially, which could influence activation measures.